Although significant advances in graphical user interfaces have made software applications more accessible and productive for personal computer users, resulting increased ease of use has fostered a demand to include more and more sophisticated features. When first exposed to a complicated software application, a new user can be confused by an overwhelming number and complexity of features accessible through menus and toolbars of a user interface. It is not uncommon for a user unfamiliar with the software features to resort to menu surfing, rapidly switching from menu item to menu item, in the hope of discovering how to effect a desired feature. Even when users know one or more ways to perform a task in a software application, they may not realize an efficient approach. Also, while users may know how to perform tasks, instead of performing the task themselves, they would rather have a system proactively determine their needs and perform or offer to perform operations, such as launching applications that will soon be needed, exiting applications, and prefetching files or information from distant servers to make such files or information available more efficiently when they are ultimately requested.
Determining in an automated manner the best actions to perform or the best information to provide users, while they work with software, requires the development of computational methods that operate in conjunction with software programs and that have an ability to identify a user's needs, intentions, or goals from aspects of that user's background and actions. Such methods may be derived from construction and use of models that explicitly links the users'needs to their backgrounds and actions.
There have been several studies of the use of models for reasoning about the intentions of people performing a variety of tasks, and for making available advice or assistance based on these models. Researchers studying the use of computers in education have attempted to construct programs that use models of users to determine the source of misunderstanding and the best way to tutor a student. These models can look at the answers users give to questions and other challenges. Strategic Computing Initiative of the 1980s focused in part on methods for doing reasoning about a pilot's intentions from a diverse set of inputs, and for providing them with the best assistance with piloting and warfighting with fighter jets. Within many such "pilot associates" projects, engineers attempted to design systems that could reason about the intentions of users. Typically, rule-based or logical approaches were used for determining or prioritizing options.
There has also been some work in the use of probabilistic models in the modeling of user's needs for such tasks as monitoring complex, time-critical applications. Early discussion of probabilistic dependency models, with applications to enhancing the display of information, occurred in projects experimenting with user modeling for controlling information displayed to aircraft pilots. Probabilistic inference was explored as a means for controlling the information being displayed to pilots. Other work included using decision-theoretic methods to reason about most important information to display to flight engineers at NASA Mission Control Center. The methods consider context, user expertise level, and telemetry from the Space Shuttle to make these decisions, and take advantage of time-criticality and probabilistic models of user beliefs and intentions. Other related work has explored the best kind of information and explanations to display to pilots given various tradeoffs in display. In other work, researchers have explored the use of probabilistic models in assisting computer users working with command line input systems of a computer operating system.
There have been several projects focusing on the use of sets of logical rules and heuristics, based on logical rules, for providing users with assistance based on the context and activity. For instance, common context-sensitive help that is made available in applications written for a Microsoft Windows operating system is accessed by selecting a graphical object and clicking a standard help command (e.g., a function key on the keyboard). Help information is then provided which describes the functionality of the graphical object such a toolbar. Another example is a TipWizard system in Microsoft Excel products which employs a large set of logical rules that continue to scan for sequences of user events and provide assistance to users on performing tasks more efficiently. In work at Apple Computer described by U.S. Pat. No. 5,390,281 to Luciw et al., sets of rules are used for providing assistance to users on tasks by employing a heuristic of comparing a number of conditions for a rule to fire with a specific number of conditions that are seen in a particular situation to yield a fraction and for interpreting this fraction as a strength associated with the action indicated by the rule.
There has also been work on more sophisticated probabilistic models that consider a uncertainty in user goals and needs for software assistance. An Answer Wizard feature of Microsoft Office products utilizes probabilistic models for interpreting words in a user's free-text query with a focus on assisting users with using computer software applications. In this work, an uncertain relationship between sets of related words that might be seen in a query and various kinds of assistance are modeled with probability. This work has been described in U.S. patent application Ser. No. 08/400,797; filed Mar. 7, 1995; now U.S. Pat. No. 5,694,559.
However, none of these former efforts has employed sophisticated methods for reasoning under uncertainty about a user's needs for assistance by considering multiple pieces of evidence, including information about a user's background and a user's activities that are autonomously sensed by the program, and, then, has offered to provide relevant assistance to a user based on this reasoning. Nor have these efforts considered monitoring user interaction with a software program so that, when a user does inquire about help directly with free-text queries, a help feature combines the analysis of the words in the free-text query with the ongoing analysis of user actions and program context.
What is desirable is an intelligent user assistance facility that autonomously senses that the user may need assistance in using a particular feature or to accomplish a specific task, and that offers to provide relevant assistance based upon considering multiple pieces of evidence, including information about the user's background and the user's activities. What is also desirable is an intelligent user assistance facility that combines the analysis of the words in the free-text query with the ongoing analysis of user actions and program context whenever a user asks directly for help with free-text queries. This invention solves the problem of autonomously sensing that a user may need assistance in using a particular feature or to accomplish a specific task, and offering to provide relevant assistance based upon considering multiple pieces of evidence, including information about a user's background and a user's activities. This invention also solves the problem of combining the analysis of the words in the free-text query with the ongoing analysis of user actions and program context whenever a user asks directly for help with free-text queries. In addition, the invention is able to monitor and perform inference about several classes of events, including a state of key data structures in a program, general sequences of user inputs with a mouse-controlled cursor in a normal course of interacting with a graphical user interface, words typed in free-text queries for assistance, visual information about users, such as gaze and gesture information, and speech information.